{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MNIST Dataset & Database\n",
    "\n",
    "In the [MNIST tutorial](https://github.com/caffe2/caffe2/blob/master/caffe2/python/tutorials/MNIST.ipynb) we use a lmdb database. You can also use leveldb or even minidb by changing the type reference when you get ready to read from the db's.\n",
    "\n",
    "\n",
    "## Dataset:\n",
    "\n",
    "You can download the raw [MNIST dataset](https://download.caffe2.ai/datasets/mnist/mnist.zip), g/unzip the dataset and labels, and make the database yourself. \n",
    "\n",
    "\n",
    "## Databases:\n",
    "\n",
    "We provide a few database formats for you to try with the MNIST tutorial. The default is lmdb. \n",
    "\n",
    "* [MNIST-nchw-lmdb](https://download.caffe2.ai/databases/mnist-lmdb.zip) - contains both the train and test lmdb MNIST databases in NCHW format\n",
    "* [MNIST-nchw-leveldb](https://download.caffe2.ai/databases/mnist-leveldb.zip) - contains both the train and test leveldb MNIST databases in NCHW format\n",
    "* [MNIST-nchw-minidb](https://download.caffe2.ai/databases/mnist-minidb.zip) - contains both the train and test minidb MNIST databases in NCHW format\n",
    "\n",
    "\n",
    "## Tools:\n",
    "\n",
    "### make_mnist_db\n",
    "\n",
    "If you like LevelDB you can use Caffe2's `make_mnist_db` binary to generate leveldb databases. This binary is found in `/caffe2/build/caffe2/binaries/` or depending on your OS and installation, in `/usr/local/bin/`.\n",
    "\n",
    "Here is an example call to `make_mnist_db`:\n",
    "\n",
    "```\n",
    "./make_mnist_db --channel_first --db leveldb --image_file ~/Downloads/train-images-idx3-ubyte --label_file ~/Downloads/train-labels-idx1-ubyte --output_file ~/caffe2/caffe2/python/tutorials/tutorial_data/mnist/mnist-train-nchw-leveldb\n",
    "\n",
    "./make_mnist_db --channel_first --db leveldb --image_file ~/Downloads/t10k-images-idx3-ubyte --label_file ~/Downloads/t10k-labels-idx1-ubyte --output_file ~/caffe2/caffe2/python/tutorials/tutorial_data/mnist/mnist-test-nchw-leveldb\n",
    "```\n",
    "Note leveldb can get deadlocked if more than one user attempts to open the leveldb at the same time. This is why there is logic in the Python below to delete LOCK files if they're found.\n",
    "\n",
    "TODO: it would be great to extend this binary to create other database formats \n",
    "\n",
    "### Python script\n",
    "\n",
    "You can use the Python in the code block below to download the dataset with `DownloadResource`, call the `make_mnist_db` binary, and generate your database with `GenerateDB`. \n",
    "\n",
    "The `DownloadResource` function can also download and extract a database for you.\n",
    "\n",
    "\n",
    "**Downloads and extracts the MNIST dataset**\n",
    "The sample function below will download and extract the dataset for you.\n",
    "```python\n",
    "DownloadResource(\"http://download.caffe2.ai/datasets/mnist/mnist.zip\", data_folder)\n",
    "```\n",
    "\n",
    "**Downloads and extracts the lmdb databases of MNIST images - both test and train**\n",
    "```python\n",
    "DownloadResource(\"http://download.caffe2.ai/databases/mnist-lmdb.zip\", data_folder)\n",
    "```\n",
    "\n",
    "**(Re)generate the leveldb database (it can get locked with multi-user setups or abandoned threads)**\n",
    "Requires the download of the dataset (mnist.zip) - see above.\n",
    "\n",
    "```python\n",
    "GenerateDB(image_file_train, label_file_train, \"mnist-train-nchw-leveldb\")\n",
    "GenerateDB(image_file_test, label_file_test, \"mnist-test-nchw-leveldb\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "def DownloadResource(url, path):\n",
    "    '''Downloads resources from s3 by url and unzips them to the provided path'''\n",
    "    import requests, zipfile, StringIO\n",
    "    print(\"Downloading... {} to {}\".format(url, path))\n",
    "    r = requests.get(url, stream=True)\n",
    "    z = zipfile.ZipFile(StringIO.StringIO(r.content))\n",
    "    z.extractall(path)\n",
    "    print(\"Completed download and extraction.\")\n",
    "\n",
    "    \n",
    "def GenerateDB(image, label, name):\n",
    "    '''Calls the make_mnist_db binary to generate a leveldb from a mnist dataset'''\n",
    "    name = os.path.join(data_folder, name)\n",
    "    print 'DB: ', name\n",
    "    if not os.path.exists(name):\n",
    "        syscall = \"/usr/local/bin/make_mnist_db --channel_first --db leveldb --image_file \" + image + \" --label_file \" + label + \" --output_file \" + name\n",
    "        # print \"Creating database with: \", syscall\n",
    "        os.system(syscall)\n",
    "    else:\n",
    "        print \"Database exists already. Delete the folder if you have issues/corrupted DB, then rerun this.\"\n",
    "        if os.path.exists(os.path.join(name, \"LOCK\")):\n",
    "            # print \"Deleting the pre-existing lock file\"\n",
    "            os.remove(os.path.join(name, \"LOCK\"))\n",
    "\n",
    "            \n",
    "current_folder = os.path.join(os.path.expanduser('~'), 'caffe2_notebooks')\n",
    "data_folder = os.path.join(current_folder, 'tutorial_data', 'mnist')\n",
    "\n",
    "# Downloads and extracts the lmdb databases of MNIST images - both test and train\n",
    "if not os.path.exists(os.path.join(data_folder,\"mnist-train-nchw-lmdb\")):\n",
    "    DownloadResource(\"http://download.caffe2.ai/databases/mnist-lmdb.zip\", data_folder)\n",
    "\n",
    "# Downloads and extracts the MNIST data set\n",
    "if not os.path.exists(os.path.join(data_folder, \"train-images-idx3-ubyte\")):\n",
    "    DownloadResource(\"http://download.caffe2.ai/datasets/mnist/mnist.zip\", data_folder)\n",
    "\n",
    "# (Re)generate the leveldb database (it can get locked with multi-user setups or abandoned threads)\n",
    "# Requires the download of the dataset (mnist.zip) - see DownloadResource above.\n",
    "# You also need to change references in the MNIST tutorial code where you train or test from lmdb to leveldb\n",
    "image_file_train = os.path.join(data_folder, \"train-images-idx3-ubyte\")\n",
    "label_file_train = os.path.join(data_folder, \"train-labels-idx1-ubyte\")\n",
    "image_file_test = os.path.join(data_folder, \"t10k-images-idx3-ubyte\")\n",
    "label_file_test = os.path.join(data_folder, \"t10k-labels-idx1-ubyte\")\n",
    "GenerateDB(image_file_train, label_file_train, \"mnist-train-nchw-leveldb\")\n",
    "GenerateDB(image_file_test, label_file_test, \"mnist-test-nchw-leveldb\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Code Changes for Other DBs\n",
    "\n",
    "If you chose to use a format other than lmdb you will need to change a couple lines of code. When you use `ModelHelper` to instantiate the CNN, you pass in the `db` parameter with a path and the `db_type` with the type of db. You would need to update both of these values. Since you create two networks, one for training and one for testing, you would need to update the code for both of these.\n",
    "\n",
    "**Default code using lmdb**\n",
    "```python\n",
    "train_model = model_helper.ModelHelper(name=\"mnist_train\", arg_scope=arg_scope)\n",
    "data, label = AddInput(\n",
    "    train_model, batch_size=64,\n",
    "    db=os.path.join(data_folder, 'mnist-train-nchw-lmdb'),\n",
    "    db_type='lmdb')\n",
    "```\n",
    "\n",
    "**Updated code using leveldb**\n",
    "```python\n",
    "train_model = model_helper.ModelHelper(name=\"mnist_train\", arg_scope=arg_scope)\n",
    "data, label = AddInput(\n",
    "    train_model, batch_size=64,\n",
    "    db=os.path.join(data_folder, 'mnist-train-nchw-leveldb'),\n",
    "    db_type='leveldb')\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
